Multi-input multi-output time encoding and decoding machines

ABSTRACT

Methods and systems for encoding and decoding signals using a Multi-input Multi-output Time Encoding Machine (TEM) and Time Decoding Machine are disclosed herein.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation of Ser. No. 12/645,292, filed Dec.22, 2009, which is a continuation of International ApplicationPCT/US2008/068790 filed Jun. 30, 2008, which claims priority from: U.S.patent application Ser. No. 11/965,337 filed on Dec. 27, 2007; U.S.Provisional Patent Application No. 60/946,918 filed on Jun. 28, 2007;and U.S. Provisional Patent Application No. 61/037,224 filed on Mar. 17,2008, the entire disclosures of which are explicitly incorporated byreference herein.

STATEMENT REGARDING FEDERALLY SPONSORED RESEARCH

This invention was made with government support under grantsCCF-06-35252, awarded by the National Science Foundation, and R01DC008701-01, awarded by the National Institutes of Health. Thegovernment has certain rights in the invention.

BACKGROUND

1. Field

The present application relates to methods and systems for a Multi-inputMulti-output (MIMO) Time Encoding Machines (TEMs) and Time DecodingMachines (TDMs) as well as uses of TEMs and TDMs for encoding anddecoding video signals.

2. Background Art

Most signals in the natural world are analog, i.e., cover a continuousrange of amplitude values. However, most computer systems for processingthese signals are binary digital systems. Generally, synchronousanalog-to-digital (A/D) converters are used to capture analog signalsand present a digital approximation of the input signal to a computerprocessor. That is, at precise moments in time synchronized to a systemclock, the amplitude of the signal of interest is captured as a digitalvalue. When sampling the amplitude of an analog signal, each bit in thedigital representation of the signal represents an increment of voltage,which defines the resolution of the A/D converter. Analog-to-digitalconversion is used in numerous applications, such as communicationswhere a signal to be communicated can be converted from an analogsignal, such as voice, to a digital signal prior to transport along atransmission line.

Applying traditional sampling theory, a band limited signal can berepresented with a quantifiable error by sampling the analog signal at asampling rate at or above what is commonly referred to as the Nyquistsampling rate. It is a continuing trend in electronic circuit design toreduce the available operating voltage provided to integrated circuitdevices. In this regard, power supply voltages for circuits areconstantly decreasing. While digital signals can be processed at thelower supply voltages, traditional synchronous sampling of the amplitudeof a signal becomes more difficult as the available power supply voltageis reduced and each bit in the A/D or D/A converter reflects asubstantially lower voltage increment.

Time Encoding Machines (TEMs) can encode analog information in the timedomain using only asynchronous circuits. Representation in the timedomain can be an alternative to the classical sampling representation inthe amplitude domain. Applications for TEMs can be found in low powernano-sensors for analog-to-discrete (A/D) conversion as well as inmodeling olfactory systems, vision and hearing in neuroscience.

SUMMARY

Systems and methods for using MIMO TEMs and TDMs are disclosed herein.

According to some embodiments of the disclosed subject matter, methodsfor encoding a plurality (M) of components of a signal include filteringeach of the M components into a plurality (N) filtered signals andencoding each of the N filtered-signals using at least one Time EncodingMachine (TEM) to generate a plurality (N) of TEM-encoded filteredsignals. In some embodiments include the TEM can be anintegrate-and-fire neuron, can have multiplicative coupling, and/or canbe an asynchronous sigma/delta modulator. In some embodiments, a biasvalue can be added to each of the N filtered-signals. In someembodiments, the M signals can be irregularly sampled.

In further embodiments, each of the N filtered signals can berepresented by the equation v^(j)=(h^(j))^(T)*u, where h^(j)=[h^(j1),h^(j2), . . . h^(jM)]^(T) is a filtering vector corresponding to one ofthe N filtered signals represented by j. In some embodiments, the NTEM-encoded filtered signals can represented by q_(k) ^(j), where q_(k)^(j)=κ^(j)δ^(j)−b^(j)(t_(k+1) ^(j)−t_(k) ^(j)), for all timesrepresented by a value k ∈

, and each of the N filtered signals represented by a value j, j=1, 2, .. . N, where κ^(j) is an integration constant, δ^(j) is a thresholdvalue, and b^(j) is a bias value for each of the N filtered signals.

According to some embodiments of the disclosed subject matter, methodsfor decoding a TEM-encoded signal include, receiving a plurality (N) ofTEM-encoded filtered signals, decoding the N TEM-encoded filteredsignals using at least one Time Decoding Machine (TDM) to generate aplurality (N) of TDM-decoded signal components, and filtering each ofthe N TDM-decoded signal components into a plurality (M) of outputsignals components. In some embodiments, one of the M output signalcomponents, represented by the i-th component of the vector valuedsignal u, |u^(i)(t)|≦c^(i), can be recovered by solving for

${{u^{i}(t)} = {\sum\limits_{j = 1}^{N}{\sum\limits_{k \in }{c_{k}^{j}{\psi_{k}^{ji}(t)}}}}},{{{where}\mspace{14mu} {\psi_{k}^{ji}(t)}} = {\left( {{\overset{\sim}{h}}^{ji}*g} \right)\left( {t - s_{k}^{j}} \right)}},$

for all i, i=1, 2, . . . M, s_(k) ^(j)=(t_(k+1) ^(j)+t_(k) ^(j))/2,{tilde over (h)}^(ji) is the involution of h^(ji), h^(ji) is representedin a TEM-filterbank

${{h(t)} = \begin{bmatrix}{h^{11}(t)} & {h^{12}(t)} & \ldots & {h^{1M}(t)} \\{h^{21}(t)} & {h^{22}(t)} & \ldots & {h^{2M}(t)} \\\vdots & \vdots & \ddots & \vdots \\{h^{N\; 1}(t)} & {h^{N\; 2}(t)} & \ldots & {h^{NM}(t)}\end{bmatrix}},$

and [c^(j)]_(k)=c_(k) ^(j) j=1, 2, . . . N, where c=[c¹, c², . . . ,c^(N)]^(T), c=G⁺q, where q=[q¹, q², . . . q^(N)]^(T) and[q^(j)]_(k)=q_(k) ^(j) and

$\left\lbrack G^{ij} \right\rbrack_{kl} = {\sum\limits_{m = 1}^{M}{\int_{c_{k}^{i}}^{c_{k + 1}^{i}}{h^{im}*{\overset{\sim}{h}}^{jm}*{g\left( {t - s_{l}^{j}} \right)}{{s}.}}}}$

In some embodiments, the M TEM-encoded signals can be irregularlysampled.

According to some embodiments of the disclosed subject matter, methodsfor encoding a video stream signal include filtering the video streamsignal into a plurality (N) of spatiotemporal field signals, andencoding each of the N spatiotemporal field signals with a Time EncodingMachine to generate a plurality (N) of TEM-encoded spatiotemporal fieldsignals. In some embodiments, the spatiotemporal field signals can bedescribed by an equation: v^(j)(t)=∫_(−∞) ^(+∞)(∫∫_(X)D^(j)(x, y, s)I(x,y, t−s)dxdy)ds, where D^(j)(x, y, s) is a filter function, and I(x, y,t) represents the input video stream.

In some embodiments, the N TEM-encoded spatiotemporal field signals canbe represented by a sampling function: ψ_(k) ^(j)(x, y, t)=D(x,y,−t)*g(t−s_(k) ^(j)), for k spike times, for each (x, y) in a boundedspatial set, where j corresponds to each of the N TEM-encodedspatiotemporal field signals, and where g(t)=sin(Ωt)/πt.

According to some embodiments of the disclosed subject matter, methodsfor decoding a TEM-encoded video stream signal include receiving aplurality (N) of TEM-encoded spatiotemporal field signals, decoding eachof the N TEM-encoded spatiotemporal field signals using a Time DecodingMachine (TDM) to generate a TDM-decoded spatiotemporal field signal, andcombining each of the TDM-decoded spatiotemporal field signals torecover the video stream signal.

In some embodiments, the decoding and combining can be achieved byapplying an equation:

${{I\left( {x,y,t} \right)} = {\sum\limits_{j = 1}^{N}{\sum\limits_{k \in }{c_{k}^{j}{\psi_{k}^{j}\left( {x,y,t} \right)}}}}},$

where ψ_(k) ^(j)(x, y, t)=D(x, y, t)*g(t−s_(k) ^(j)), for k spike times,for each (x, y) in a bounded spatial set, where j corresponds to each ofthe N TEM-encoded spatiotemporal field signals, and whereg(t)=sin(Ωt)/πt, and where [c^(j) _(k)=c_(k) ^(j) and c=[c¹, c², . . .c^(J)]^(T), c=G⁺q, where T denotes a transpose, q=[q¹, q², . . .q^(N)]^(T), [q^(j)]_(k)=q_(k) ^(j) and G⁺ denotes a pseudoinverse, amatrix G is represented by

${G = \begin{bmatrix}G^{11} & G^{12} & \ldots & G^{1N} \\G^{21} & G^{22} & \ldots & G^{2N} \\\vdots & \vdots & \ddots & \vdots \\G^{N\; 1} & G^{N\; 2} & \ldots & G^{NN}\end{bmatrix}},$

and [G^(ij)]_(kl)=<D^(t)(x, y, •)*g(•−t_(k) ^(i)), D^(j)(x, y,•)*g(•−t_(l) ^(j)).

According to some embodiments of the disclosed subject matter, methodsof altering a video stream signal include receiving a plurality (N) ofTEM-encoded spatiotemporal field signals from a plurality (N) ofTEM-filters applying a switching matrix to map the N TEM-encodedspatiotemporal field signals to a plurality (N) of reconstructionfilters in a video stream signal TDM. In some embodiments for rotatingthe video stream signal the switching matrix can map each of the NTEM-encoded spatiotemporal field signals from a TEM-filter ([x,y], α, θ)to a reconstruction filter ([x,y], α, θ+lθ₀), where lθ₀ represents adesired value of rotation.

In some embodiments for zooming the video stream signal the switchingmatrix can map each of the N TEM-encoded spatiotemporal field signalsfrom a TEM-filter ([x,y], α, θ) to a reconstruction filter ([x,y], α₀^(m)α, θ), where α₀ ^(m) represents a desired value of zoom.

In some embodiments for translating the video stream signal by a value[nb₀, kb₀] the switching matrix can map each of the N TEM-encodedspatiotemporal field signals from a TEM-filter ([x,y], α, θ) to areconstruction filter at ([x+nb₀, y+kb₀], α, θ).

In some embodiments for zooming the video stream signal by a value α₀^(m) and translating the video stream signal by a value [nb₀, kb₀] theswitching matrix can map each of the N TEM-encoded spatiotemporal fieldsignals from a TEM-filter ([x,y], α, θ) to a reconstruction filter at([x+α₀ ^(m)nb₀, y+α₀ ^(m)kb₀], α₀ ^(m)α, θ).

According to some embodiments of the disclosed subject matter, methodsof encoding a video signal include inputting the video signal into afirst and second time encoding machine (TEM), the first TEM including afirst TEM-input and a first TEM-output, the second TEM including asecond TEM-input and a second TEM-output, wherein the first TEM-outputis connected to the first TEM-input and the second TEM-input to providenegative feedback and the second TEM-output is connected to the firstTEM-input and the second TEM-input to provide positive feedback.

Some embodiments further include outputting a first set of triggervalues from the first TEM according to an equation

${u\left( t_{k}^{1} \right)} = {{{+ \delta^{1}} + {\sum\limits_{l < k}{h^{11}\left( {t_{k}^{1} - t_{l}^{1}} \right)}} - {\sum\limits_{l}{{h^{21}\left( {t_{k}^{1} - t_{l}^{2}} \right)}1_{\{{t_{l}^{2} < t_{k}^{1}}\}}}}} = q_{k}^{1}}$

and outputting a second set of trigger values from the second TEMaccording to an equation

${u\left( t_{k}^{2} \right)} = {{{- \delta^{2}} + {\sum\limits_{l < k}{h^{22}\left( {t_{k}^{2} - t_{l}^{2}} \right)}} - {\sum\limits_{l}{{h^{12}\left( {t_{k}^{2} - t_{l}^{1}} \right)}1_{\{{t_{l}^{1} < t_{k}^{2}}\}}}}} = {q_{k}^{2}.}}$

Yet other embodiments further include outputting a first set of triggervalues from the first TEM according to an equation

${\int_{c_{k}^{1}}^{c_{k + 1}^{1}}{{u(s)}{s}}} = {{\kappa^{1}\delta^{1}} - {b^{1}\left( {t_{k + 1}^{1} - t_{k}^{1}} \right)} + {\sum\limits_{l < k}{\int_{c_{k}^{1}}^{c_{k + 1}^{1}}{h^{11}\left( {s - t_{l}^{1}} \right)}}} - {\sum\limits_{l}{\int_{c_{k}^{1}}^{c_{k + 1}^{1}}{{h^{21}\left( {s - t_{l}^{2}} \right)}{s}\; 1_{\{{t_{l}^{2} < t_{k}^{1}}\}}}}}}$

and outputting a second set of trigger values from the second TEMaccording to an equation

${\int_{t_{k}^{2}}^{t_{k + 1}^{2}}{{u(s)}{s}}} = {{\kappa^{2}\delta^{2}} - {b^{2}\left( {t_{k + 1}^{1} - t_{k}^{1}} \right)} + {\sum\limits_{l < k}{\int_{t_{k}^{2}}^{t_{k + 1}^{2}}{h^{22}\left( {s - t_{l}^{2}} \right)}}} - {\sum\limits_{l}{\int_{t_{k}^{2}}^{t_{k + 1}^{2}}{{h^{12}\left( {s - t_{l}^{1}} \right)}{s}\; {1_{\{{t_{l}^{1} < t_{k}^{2}}\}}.}}}}}$

According to some embodiments of the disclosed subject matter, methodsof decoding a video signal include receiving first and secondTEM-encoded signals and applying an equation

${{u(t)} = {{\sum\limits_{k \in }{c_{k}^{1}{\psi_{k}^{1}(t)}}} + {\sum\limits_{k \in }{c_{k}^{2}{\psi_{k}^{2}(t)}}}}},$

where ψ_(k) ^(j)(t)=g(t−t_(k) ^(j)), for j=1, 2, g(t)=sin(Ωt)/πt, t ∈

, c=[c¹; c²] and [c^(j)]_(k)=c_(k) ^(j), and a vector of coefficients ccan be computed as c=G⁺q, where q=[q¹; q²] with [q^(j)]_(k)=q_(k) ^(j)and

${G = \begin{bmatrix}G^{11} & G^{12} \\G^{21} & G^{22}\end{bmatrix}},{\left\lbrack G^{ij} \right\rbrack_{kl} = {\langle{x_{k}^{i},\psi_{l}^{j}}\rangle}},$

for all j=1, 2, and k, l ∈

.

The accompanying drawings, which are incorporated and constitute part ofthis disclosure, illustrate preferred embodiments of the disclosedsubject matter and serve to explain its principles.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 depicts a multiple input multiple output Time Encoding Machinearchitecture in accordance with some embodiments of the disclosedsubject matter;

FIG. 2 depicts a multiple input multiple output Time Decoding MachineArchitecture in accordance with some embodiments of the disclosedsubject matter;

FIG. 3 depicts a multiple input multiple output irregular samplingarchitecture in accordance with some embodiments of the disclosedsubject matter;

FIG. 4 depicts a video stream encoding device in accordance with someembodiments of the disclosed subject matter;

FIG. 5 depicts a video stream decoding device in accordance with someembodiments of the disclosed subject matter;

FIG. 6 depicts a second video stream decoding device in accordance withsome embodiments of the disclosed subject matter;

FIG. 7 depicts a video stream encoding and decoding device in accordancewith some embodiments of the disclosed subject matter;

FIG. 8 depicts a time encoding circuit in accordance with someembodiments of the disclosed subject matter;

FIG. 9 depicts a single neuron encoding circuit that includes anintegrate-and-fire neuron with feedback in accordance with someembodiments of the disclosed subject matter;

FIG. 10 depicts two interconnected ON-OFF neurons each with its ownfeedback in accordance with some embodiments of the disclosed subjectmatter; and

FIG. 11 depicts two interconnected ON-OFF neurons each with its ownfeedback in accordance with some embodiments of the disclosed subjectmatter.

Throughout the drawings, the same reference numerals and characters,unless otherwise stated, are used to denote like features, elements,components or portions of the illustrated embodiments. Moreover, whilethe present disclosed subject matter will now be described in detailwith reference to the Figs., it is done so in connection with theillustrative embodiments.

DETAILED DESCRIPTION

Improved systems, methods, and applications of Time Encoding andDecoding machines are disclose herein.

Asynchronous Sigma/Delta modulators as well as FM modulators can encodeinformation in the time domain as described in “Perfect Recovery andSensitivity Analysis of Time Encoded Bandlimited Signals” by A. A. Lazarand L. T. Toth (IEEE Transactions on Circuits and Systems-I: RegularPapers, 51(10):2060-2073, October 2004), which is incorporated byreference. More general TEMs with multiplicative coupling, feedforwardand feedback have also been characterized by A. A. Lazar in “TimeEncoding Machines with Multiplicative Coupling, Feedback andFeedforward” (IEEE Transactions on Circuits and Systems II: ExpressBriefs, 53(8):672-676, August 2006), which is incorporated by reference.TEMs realized as single and as a population of integrate-and-fireneurons are described by A. A. Lazar in “Multichannel Time Encoding withIntegrate-and-Fire Neurons” (Neurocomputing, 65-66:401-407, 2005) and“Information Representation with an Ensemble of Hodgkin-Huxley Neurons”(Neurocomputing, 70:1764-1771, June 2007), both of which areincorporated by reference. Single-input single-output (SIMO) TEMs aredescribed in “Faithful Representation of Stimuli with a Population ofIntegrate-and-Fire Neurons” by A. A. Lazar and E. A. Pnevmatikakis(Neural Computation), which is incorporated by reference.

Multi-input multi-output (MIMO) TEMs can encode M-dimensionalbandlimited signals into N-dimensional time sequences. A representationof M-dimensional bandlimited signals can be generated using anN×M-dimensional filtering kernel and an ensemble of N integrate-and-fireneurons. Each component filter of the kernel can receive input from oneof the M component inputs and its output can be additively coupled to asingle neuron. (While the embodiments described refer to “neurons,” itwould be understood by one of ordinary skill that other kinds of TEMscan be used in place of neurons.)

As depicted in FIG. 1, in one embodiment, M components of a signal 101,can be filtered by a set of filters 102. The outputs of the filters canbe additively coupled to a set of N TEMs 104 to generated a set of Nspike-sequences or trigger times 105.

Let Ξ be the set of bandlimited functions with spectral support in [−Ω,Ω]. The functions u^(i)=u^(i)(t), t ∈

, in Ξ can model the M components of the input signal. Without any lossof generality the signal components (u¹, u², . . . u^(M)) can have acommon bandwidth Ω. Further, it can be assumed that the signals in Ξhave finite energy, or are bounded with respect to the L² norm. Thus Ξcan be a Hilbert space with a norm induced by the inner product in theusual fashion.

Ξ^(M) can denote the space of vector valued bandlimited functions of theform u=[u¹, u², . . . , u^(M)]^(T), where T denotes the transpose. Ξ^(M)can be a Hilbert space with inner product defined by

$\begin{matrix}{{\langle{u,v}\rangle}_{\Xi^{M}} = {\sum\limits_{i = 1}^{M}{\langle{u^{i},v^{i}}\rangle}_{\Xi}}} & (1)\end{matrix}$

and with norm given by

$\begin{matrix}{{{u}_{\Xi^{M}}^{2} = {\sum\limits_{i = 1}^{M}{u^{\prime}}_{\Xi}^{2}}},} & (2)\end{matrix}$

where u=[u¹, u², . . . , u^(M)]^(T) ∈ Ξ^(M) and v=[v¹, v², . . . ,v^(M)]^(T) ∈ Ξ^(M).

We can let H:

^(N×M) be a filtering kernel 102 defined as:

$\begin{matrix}{{H(t)} = \begin{bmatrix}{h^{11}(t)} & {h^{12}(t)} & \ldots & {h^{1M}(t)} \\{h^{21}(t)} & {h^{22}(t)} & \ldots & {h^{2M}(t)} \\\vdots & \vdots & \ddots & \vdots \\{h^{N\; 1}(t)} & {h^{N\; 2}(t)} & \ldots & {h^{NM}(t)}\end{bmatrix}} & (3)\end{matrix}$

where it can be assumed that supp(ĥ^(ij))⊃[−Ω,Ω], for all i, i=1, 2, . .. , M and all j, j=1, 2, . . . N (supp denoting the spectral support and̂ denotes the Fourier transform). Filtering the signal u with the kernelH can lead to an N-dimensional vector valued signal v defined by:

$\begin{matrix}{{v\overset{\Delta}{=}{{H*u} = \begin{bmatrix}{\sum\limits_{i = 1}^{M}{h^{1i}*u^{i}}} \\{\sum\limits_{i = 1}^{M}{h^{2i}*u^{i}}} \\\vdots \\{\sum\limits_{i = 1}^{M}{h^{Ni}*u^{i}}}\end{bmatrix}}},} & (4)\end{matrix}$

where * denotes the convolution. Equation (4) can be written as

v ^(j)=(h ^(j))^(T) *u, j, j=1, 2, . . . N   (5)

where h^(j)=[h^(j1), h^(j2), . . . h^(jM)]^(T) is the filtering vectorof the neuron, or TEM, j, j=1, 2, . . . N. A bias b^(j) 103 can be addedto the component v^(j) of the signal v and the sum can be passed throughan integrate-and-fire neuron with integration constant κ^(j) andthreshold δ^(j) for all j, j=1, 2, . . . , N. The value (t_(k) ^(j)), k∈

can be the sequence of trigger (or spike) times 105 generated by neuronj, j=1, 2, . . . , N. In sum, the TEM depicted in FIG. 1 can map theinput bandlimited vector u into the vector time sequence (t_(k) ^(j)), k∈

, j=1, 2, . . . , N.

In some embodiments the t-transform can describe the input/outputrelationship of the TEM, or the mapping of the stimulus u(t), t ∈

, into the output spike sequence (t_(k) ^(j)), k ∈

, j=1, 2, . . . , N. The t-transform for the j-th neuron can be writtenas

$\begin{matrix}{{{\int_{t_{k}^{j}}^{t_{k + 1}^{j}}{\left( {{v^{j}(s)} + b^{j}} \right)\ {s}}} = {\kappa^{j}\delta^{j}}},{or}} & (6) \\{{{\sum\limits_{i = 1}^{M}{\int_{t_{k}^{j}}^{t_{k + 1}^{j}}{\left( {h^{ji}*u^{i}} \right)(s){s}}}} = q_{k}^{j}},} & (7)\end{matrix}$

where q_(k) ^(j)=κ^(j)δ^(j)−b^(j)(t_(k+1) ^(j)−t_(k) ^(j)), for all k ∈

, and j=1, 2, . . . N.

In some embodiments, as depicted in FIG. 2, recovering the stimuli orsignal that was encoded 105 can be achieved by seeking the inverse ofthe t-transform 201. We can let g(t)=sin(Ωt)/πt, t ∈

, be the impulse response of a low pass filter (LPF) with cutofffrequency at Ω. From Equation (5) v^(j) ∈ Ξ and therefore thet-transform defined by Equation (7) can be written in an inner-productform as:

$\begin{matrix}{{\sum\limits_{i = 1}^{M}{\langle{{h^{ji}*u^{i}},{g*1_{\lbrack{t_{k}^{j},t_{k + 1}^{j}}\rbrack}}}\rangle}} = \left. q_{k}^{j}\Leftrightarrow{or} \right.} & (8) \\{{\sum\limits_{i = 1}^{M}{\langle{u^{i},{{\overset{\sim}{h}}^{ji}*g*1_{\lbrack{t_{k}^{j},t_{k + 1}^{j}}\rbrack}}}\rangle}} = q_{k}^{j}} & \;\end{matrix}$

where {tilde over (h)}^(ji) is the involution of h^(ji). From Equality(8) we can say that stimulus u=(u¹, u², . . . , u^(M))^(T) can bemeasured by projecting it onto the sequence of functions ({tilde over(h)}^(j)*g*1_([t) _(k) _(j) _(,t) _(k+1) _(j) _(])), k ∈

, and j=1, 2, . . . N. The values of the measurements, q_(k) ^(j), k ∈

, and j=1, 2, . . . N, are available for recovery. Thus the TEM can actas a sampler of the stimulus u, and because the spike times can dependon the stimulus, the TEM can act as a stimulus dependent sampler.

With (t_(k) ^(j)), k ∈

, the spike times of neuron j, the t-transform, as shown in Equation(8), can be written in an inner product form as

$\begin{matrix}{q_{k}^{j} = {\langle{u,\varphi_{k}^{j}}\rangle}} & (9) \\{where} & \; \\{\varphi_{k}^{j} = {{{\overset{\sim}{h}}^{j}*g*1_{\lbrack{t_{k}^{j},t_{k + 1}^{j}}\rbrack}} = {\begin{bmatrix}\begin{matrix}\begin{matrix}{\overset{\sim}{h}}^{j\; 1} \\{\overset{\sim}{h}}^{j\; 2}\end{matrix} \\\vdots\end{matrix} \\{\overset{\sim}{h}}^{j\; M}\end{bmatrix}g*1_{\lbrack{t_{k}^{j},t_{k + 1}^{j}}\rbrack}}}} & (10)\end{matrix}$

for all k ∈

, j=1, 2, . . . N, t ∈

.

The stimulus u 101 can be recovered from Equation (9) if φ=φ_(k) ^(j),where k ∈

, j=1, 2, . . . N, is a frame for Ξ^(M). Signal recover algorithms canbe obtained using the frame ψ=(ψ_(k) ^(j)), k ∈

, j=1, 2, . . . N, where

ψ_(k) ^(j)(t)=({tilde over (h)} ^(j) *g)(t−s _(k) ^(j)),   (11)

and s_(k) ^(j)=(t_(k+1) ^(j)+t_(k) ^(j))/2.

A filtering kernel H can be said to be BIBO stable if each of thefilters h^(ji)=h^(ji)(t), t ∈

, j=1, 2, . . . N, and i=1, 2, . . . , M, is bounded-inputbounded-output stable, i.e.,

${h^{ji}}_{1}\overset{\Delta}{=}{{\int_{}^{\;}{{{h^{ji}(s)}}\ {s}}} < {\infty.}}$

Filtering vectors h^(j)=[h^(j1), h^(j2), . . . h^(jM)]^(T), j=1, 2, . .. , N, can be said to be BIBO stable if each of the components h^(ji),i=1, 2, . . . M, is BIBO stable. Further, if Ĥ:

→

^(N×M) is the Fourier transform of the filtering kernel H, then[Ĥ(w)]^(nm)=

h^(nm)(s)exp(−iws)ds, for all n=1, 2, . . . , N, and m=1, 2, . . . M,where i=√{square root over (−1)}. The filtering vectors h^(j) can besaid to have full spectral support if supp(ĥ^(ij)) 533 [−Ω, Ω], for alli, i=1, 2, . . . M. A BIBO filtering kernel H can be said to beinvertible if Ĥ has rank M for all w ∈ [−Ω, Ω]. Filtering vectors(h^(j)), j=1, 2, . . . , N, can be called linearly independent if theredo not exist real numbers a_(j), j=1, 2, . . . N, not all equal to zero,and real numbers α^(j), j=1, 2, . . . N, such that

${\sum\limits_{j = 1}^{N}{{a_{j}\left( {h^{j}*g} \right)}\left( {t - \alpha^{j}} \right)}} = 0$

for all t, t ∈

(except on a set of Lebesgue-measure zero).

Assuming that the filters h^(j)=h^(j)(t), t ∈

, are BIBO stable, linearly independent and have full spectral supportfor all j, j=1, 2, . . . N, and that matrix H is invertible, then theM-dimensional signal u=[u¹, u², . . . , u^(M)]^(T) can be recovered as

$\begin{matrix}{{{u(t)} = {\sum\limits_{j = 1}^{N}{\sum\limits_{k \in }^{\;}{c_{k}^{j}{\psi_{k}^{j}(t)}}}}},} & (12)\end{matrix}$

where c_(k) ^(j), k ∈

, j=1, 2, . . . N are suitable coefficients provided that:

$\begin{matrix}{{\sum\limits_{j = 1}^{N}\frac{b^{j}}{\kappa^{j}\delta^{j}}} \geq {M\; \frac{\Omega}{\pi}}} & (13)\end{matrix}$

and |u^(i)(t)|<c^(j), i=1, 2, . . . M.

Letting [c^(j)]_(k)=c_(k) ^(j) and c=[c¹, c², . . . c¹]^(T), thecoefficients c can be computed as

c=G ⁺ q   (14)

202 where T denotes the transpose, q=[q¹, q², . . . q^(N)]^(T),[q^(j)]_(k)=q_(k) ^(j) and G⁺ denotes the pseudoinverse. The entries ofthe matrix G can be given by

$\begin{matrix}{{G = \begin{bmatrix}G^{11} & G^{12} & \ldots & G^{1N} \\G^{21} & G^{22} & \ldots & G^{2N} \\\vdots & \vdots & \ddots & \vdots \\G^{N\; 1} & G^{N\; 2} & \ldots & G^{NN}\end{bmatrix}},\begin{matrix}{\left\lbrack G^{ij} \right\rbrack_{kl} = {\langle{\psi_{l}^{j},\varphi_{k}^{i}}\rangle}} \\{= {\sum\limits_{m = 1}^{M}{\int_{t_{k}^{i}}^{t_{k + 1}^{i}}{h^{im}*{\overset{\sim}{h}}^{jm}*{g\left( {t - s_{l}^{j}} \right)}{s}}}}}\end{matrix}} & (15)\end{matrix}$

for all i=1, 2, . . . , N, j=1, 2, . . . , N, k ∈

, l ∈

.

Assume that the filtering vectors h^(j)=h^(j)(t), are BIBO stable,linearly independent and have full spectral support for all j, j=1, 2, .. . N, and that matrix H is invertible. If Σ_(j=1) ^(N)b^(j)/κ^(j)δ^(j)diverges in N, then there can exist a number

such that for all N≧

, the vector valued signal u can be recovered as

$\begin{matrix}{{u(t)} = {\sum\limits_{j = 1}^{N}{\sum\limits_{k \in }{c_{k}^{j}{\psi_{k}^{j}(t)}}}}} & (16)\end{matrix}$

and the c_(k) ^(j), k ∈

, j=1, 2, . . . N, are given in the matrix form by c=G⁺q.

In some embodiments, the previously disclosedMultiple-Input-Multiple-Output scheme can be applied to an IrregularSampling problem, as depicted in FIG. 3. While similar to the previouslydisclosed MIMO-TEM, integrate-and-fire neurons can be replaced byirregular (amplitude) sampliers.

The samples for each signal 302 v^(j)=h^(j1)*u¹+h^(j2)*u²+ . . .+h^(jM)*u^(M) at times (s_(k) ^(j))), k ∈

, j=1, 2, . . . N, respectively can be recovered ands_(k)=(t_(k+1)+t_(k))/2, k ∈

.

As with previous embodiments, assuming that the filtering vectors h^(j)are BIBO stable for all j=1, 2, . . . N, and that H is invertible, thevector valued signal u, sampled with the circuit of FIG. 3, can berecovered as:

$\begin{matrix}{{u(t)} = {\sum\limits_{j = 1}^{N}{\sum\limits_{k \in }{c_{k}^{j}{\varphi_{k}^{j}(t)}}}}} & (17)\end{matrix}$

provided that

$D > {M\frac{\Omega}{\pi}}$

holds, where D is the total lower density. Further for [c^(j)]_(k)=c_(k)^(j) and c=[c¹, c², . . . c^(N)]^(T), the vector of coefficients c canbe computed as c=G⁺q, where q=[q¹, q², . . . , q^(N)]^(T) and[q^(j)]_(k)=q_(k) ^(j). The entries of the G matrix can be given by:

$\begin{matrix}\begin{matrix}{\left\lbrack G^{ij} \right\rbrack_{kl} = {\langle\varphi^{j^{l},\psi_{k}^{i}}\rangle}} \\{= {\sum\limits_{m = 1}^{M}{\left( {h^{im}*{\overset{\sim}{h}}^{jm}*g*1_{\lbrack{t_{j}^{j},t_{i + 1}^{j}}\rbrack}} \right){\left( s_{k}^{i} \right).}}}}\end{matrix} & (18)\end{matrix}$

In other embodiments, assuming that the filtering vectors h^(j) are BIBOstable and have full spectral support for all j=1, 2, . . . N and that His invertible, the vector valued signal u, sampled with the circuit ofFIG. 3, can be recovered as

u=Σ _(j=1) ^(N)Σ_(k ∈ ℑ) c _(k) ^(j)ψ_(k) ^(j)(t)   (19).

provided that

$D > {M\frac{\Omega}{\pi}}$

holds where D is the total lower density. Further for [c^(j)]_(k)=c_(k)^(j) and c=[c¹, c², . . . c^(N)]^(T), the vector of coefficients c canbe computed as c=G⁺q, where q=[q¹, q², . . . q^(N)]^(T) and[q^(j)]_(k)=q_(k) ^(j). The entries of the G matrix can be given by:

$\begin{matrix}{\left\lbrack G^{ij} \right\rbrack_{kl} = {{\langle{\psi_{l}^{j},\psi_{k}^{i}}\rangle} = {\sum\limits_{m = 1}^{M}{\left( {h^{im}*{\overset{\sim}{h}}^{jm}*g} \right){\left( {s_{k}^{i} - s_{l}^{j}} \right).}}}}} & (20)\end{matrix}$

In some embodiments, TEMs and TDMs can be used to encode and decodevisual stimuli such as natural and synthetic video streams, for examplemovies or animations. Encoding and decoding visual stimuli is criticalfor, among other reasons, the storage, manipulation, and transmission ofvisual stimuli, such as multimedia in the form of video. Specifically,the neuron representation model and its spike data can be used to encodevideo from its original analog format into a “spike domain” ortime-based representation; store, transmit, or alter such visual stimuliby acting on the spike domain representation; and decode the spikedomain representation into an alternate representation of the visualstimuli, e.g., analog or digital. Moreover, in “acting” on the visualstimuli through use of the spike domain, one can dilate (or zoom) thevisual stimuli, translate or move the visual stimuli, rotate the visualstimuli, and perform any other linear operation or transformation byacting or manipulating the spike domain representation of the visualstimuli.

Widely used modulation circuits such as Asynchronous Sigma/DeltaModulators and and FM modulators have been shown to be instances of TEMsby A. A. Lazar and E. A. Pnevmatikakis in “A Video Time EncodingMachine” (IEEE International Conference on Image Processing, San Diego,Calif., Oct. 12-15, 2008), which is incorporated by reference. TEMsbased on single neuron models such as integrate-and-fire (IAF) neurons,as described by A. A. Lazar in “Time Encoding with an Integrate-and-FireNeuron with a Refractory Period” (Neurocomputing, 58-60:53-58, June2004), which is incorporated by reference, and more generalHodgkin-Huxley neurons with multiplicative coupling, feedforward andfeedback have been described by A. A. Lazar in “Time Encoding Machineswith Multiplicative Coupling, Feedback and Feedforward,” which wasincorporated by reference above. Multichannel TEMs realized withinvertible filterbanks and invertible IAF neurons have been studied byA. A. Lazar in “Multichannel Time Encoding with Integrate-and-FireNeurons,” which is incorporated by reference above, and TEMs realizedwith a population of integrate-and-fire neurons have been investigatedby A. A. Lazar in “Information Representation with an Ensemble ofHodgkin-Huxley Neurons,” which was incorporated by reference above. Anextensive characterization of single-input single-output (SIMO) TEMs canbe found in A. A. Lazar and E. A. Pnevmatikakis' “FaithfulRepresentation of Stimuli with a Population of Integrate-and-FireNeurons,” which is incorporated by reference above.

FIG. 8 depicts an embodiment of a time encoding circuit. The circuit canmodel the responses of a wide variety of retinal ganglion cells (RGCs)and lateral geniculate nucleus (LGN) neurons across many differentorganisms. The neuron can fire whenever its membrane potential reaches afixed threshold δ 801. After a spike 802 is generated, the membranepotential can reset through a negative feedback mechanism 803 that getstriggered by the just emitted spike 802. The feedback mechanism 803 canbe modeled by a filter with impulse response h(t).

As previously described, TEMs can act as signal dependent samplers andencode information about the input signal as a time sequence. Asdescribed in “Perfect Recovery and Sensitivity Analysis of Time EncodedBandlimited Signals” by A. A. Lazar and László T. Tóth, which isincorporated by reference above, this encoding can be quantified withthe t-transform which describes in mathematical language the generationof the spike sequences given the input stimulus. This time encodingmechanism can be referred to as a single neuron TEM. Where (t_(k)), k ∈

, is the set of spike times of the output of the neuron, the t-transformof the TEM depicted in FIG. 1 can be written as

$\begin{matrix}{{u\left( t_{k} \right)} = {\delta + {\sum\limits_{l < k}{{h\left( {t_{k} - t_{l}} \right)}.}}}} & (21)\end{matrix}$

Equation (21) can be written in inner product form as

<u, X_(k)>=q_(k),   (22)

where

${q_{k} = {{u\left( t_{k} \right)} = {\delta + {\sum\limits_{l < k}{h\left( {t_{k} - t_{l}} \right)}}}}},$

X_(k)(t)=g(t−t_(k)), k ∈

, and g(t)=sin(Ωt)/πt, t ∈

, is the impulse response of a low pass filter with cutoff frequency Ω.The impulse response of the filter in the feedback loop can be causal,and can be decreasing with time.

FIG. 9 depicts an embodiment of a single neuron encoding circuit thatincludes an integrate-and-fire neuron with feedback. The t-transform ofthe encoding circuit can be written as

$\begin{matrix}{{{\int_{t_{k}}^{t_{k + 1}}{{u(s)}{s}}} = {{\kappa\delta} - {b\left( {t_{k + 1} - t_{k}} \right)} - {\sum\limits_{l < k}{\int_{t_{k}}^{t_{k + 1}}{{h\left( {s - t_{l}} \right)}{s}}}}}},} & (23)\end{matrix}$

or in inner product form as

<u, X_(k)>=q_(k),   (24)

with

$q_{k} = {{\kappa\delta} - {b\left( {t_{k + 1} - t_{k}} \right)} - {\sum\limits_{l < k}{\int_{t_{k}}^{t_{k + 1}}{{h\left( {s - t_{l}} \right)}{s}}}}}$

and X_(k)(t)=g*1_([t) _(k) _(,t) _(k+1) _(], for all k, k ∈)

_(.)

In some embodiments, the bandlimited input stimulus u can be recoveredas

$\begin{matrix}{{{u(t)} = {\sum\limits_{k \in }{c_{k}{\psi_{k}(t)}}}},} & (25)\end{matrix}$

where ψ_(k)(t)=g(t−t_(k)), provided that the spike density of the neuronis above the Nyquist rate Ω/π. For [c]_(k)=c_(k), the vector ofcoefficients c can be computed as c=G⁺q, where G⁺ denotes thepseudoinverse of G, [q]_(k)=q_(k) and [G]_(kl)=<X_(k), ψ_(l)>.

FIG. 10 depicts an embodiment consisting of two interconnected ON-OFFneurons each with its own feedback. Each neuron can be endowed with alevel crossing detection mechanism 1001 with a threshold δ that takes apositive value 1 and a negative value 2, respectively. Whenever a spikeis emitted, the feedback mechanism 1002 can reset the correspondingmembrane potential. In addition, each spike can be communicated to theother neuron through a cross-feedback mechanism 1003. In general, thiscross-feedback mechanism can bring the second neuron closer to itsfiring threshold and thereby increases its spike density. The two neuronmodel in FIG. 10 can mimic the ON and OFF bipolar cells in the retinaand their connections through the non-spiking horizontal cells. Thistime encoding mechanism can be described as an ON-OFF TEM.

For the TEM depicted in FIG. 10, with (t_(k) ^(j)), k ∈

representing the set of spike times of the neuron j, the t-transform ofthe ON-OFF TEM can be described by the equations

$\begin{matrix}{{{u\left( t_{k}^{1} \right)} = {{{+ \delta^{1}} + {\sum\limits_{l < k}{h^{11}\left( {t_{k}^{1} - t_{l}^{1}} \right)}} - {\sum\limits_{l}{{h^{21}\left( {t_{k}^{1} - t_{l}^{2}} \right)}1_{\{{t_{l}^{2} < t_{k}^{1}}\}}}}} = q_{k}^{1}}}{{{u\left( t_{k}^{2} \right)} = {{{- \delta^{2}} + {\sum\limits_{l < k}{h^{22}\left( {t_{k}^{2} - t_{l}^{2}} \right)}} - {\sum\limits_{l}{{h^{12}\left( {t_{k}^{2} - t_{l}^{1}} \right)}1_{\{{t_{l}^{1} < t_{k}^{2}}\}}}}} = q_{k}^{2}}},}} & (26)\end{matrix}$

for all k, k ∈

. The equations (31) can be written in inner product form as

<u, x_(k) ^(j)>=q_(k) ^(j)   (27)

for all k, k ∈

, j, j=1, 2, where x_(k) ^(j)(t)=g(t−t_(k) ^(j)), j=1, 2.

FIG. 11 depicts an embodiment consisting of two interconnected ON-OFFneurons each with its own feedback. The t-transforms of the neuronsdepicted in FIG. 11 can be described by the equations

$\begin{matrix}{{{\int_{t_{k}^{1}}^{t_{k + 1}^{1}}{{u(s)}{s}}} = {{\kappa^{1}\delta^{1}} - {b^{1}\left( {t_{k + 1}^{1} - t_{k}^{1}} \right)} + {\sum\limits_{l < k}{\int_{t_{k}^{1}}^{t_{k + 1}^{1}}{h^{11}\left( {s - t_{l}^{1}} \right)}}} - {\sum\limits_{l}{\int_{t_{k}^{1}}^{t_{k + 1}^{1}}{{h^{21}\left( {s - t_{l}^{2}} \right)}{s}\; 1_{\{{t_{l}^{2} < t_{k}^{1}}\}}}}}}}{{{\int_{t_{k}^{2}}^{t_{k + 1}^{2}}{{u(s)}{s}}} = {{\kappa^{2}\delta^{2}} - {b^{2}\left( {t_{k + 1}^{1} - t_{k}^{1}} \right)} + {\sum\limits_{l < k}{\int_{t_{k}^{2}}^{t_{k + 1}^{2}}{h^{22}\left( {s - t_{l}^{2}} \right)}}} - {\sum\limits_{l}{\int_{t_{k}^{2}}^{t_{k + 1}^{2}}{{h^{12}\left( {s - t_{l}^{1}} \right)}{s}\; 1_{\{{t_{l}^{1} < t_{k}^{2}}\}}}}}}},}} & (28)\end{matrix}$

or in inner product form as

<u, x_(k) ^(j)>=q_(k) ^(j)   (29)

with

q_(k)^(j) = ∫_(t_(k)^(j))^(t_(k + 1)^(j))u(s) s,

for all k, k ∈

, j, j=1, 2.

In some embodiments, the input stimulus u can be recovered as

$\begin{matrix}{{{u(t)} = {{\sum\limits_{k \in }{c_{k}^{1}{\psi_{k}^{1}(t)}}} + {\sum\limits_{k \in }{c_{k}^{2}{\psi_{k}^{2}(t)}}}}},} & (30)\end{matrix}$

where ψ_(k) ^(j)(t)=g(t−t_(k) ^(j)), j=1, 2, provided that the spikedensity of the TEM is above the Nyquist rate Ω/π. Moreover with c=[c¹;c²] and [c^(j)]_(k)=c_(k) ^(j), the vector of coefficients c can becomputed as c=G⁺q, where q=[q¹; q²] with [q^(j)]_(k)=q_(k) ^(j) and

${G = \begin{bmatrix}G^{11} & G^{12} \\G^{21} & G^{22}\end{bmatrix}},\mspace{14mu} {\left\lbrack G^{ij} \right\rbrack_{kl} = {\langle{x_{k}^{i},\psi_{l}^{j}}\rangle}},$

for all j=1, 2, and k, l ∈

.

In one embodiment, as depicted in FIG. 4, let

denote the space of (real) analog video streams I(x, y, t) 401 which arebandlimited in time, continuous in space, and have finite energy. Assumethat the video streams are defined on bounded spatial set X which is acompact subset of

². Bandlimited in time can mean that for every (x₀, y₀) ∈ X then I(x₀,y₀, y) ∈ Ξ, where Ξ is the space of bandlimited functions of finiteenergy. In some embodiments,

={I=I(x,y,t)|I(x₀,y₀,t) ∈ Ξ, ∀(x₀,y₀) ␣ X and I(x,y,t₀) ∈ L²(X), ∀t₀ ∈

)}. The space

, endowed with the inner product <•,•>:H

defined by:

<I ₁ ,I ₂ >=∫∫

I ₁(x,y,t)I ₂(x, y, t)dxdydt   (31)

can be a Hilbert space.

Assuming that each neuron or TEM j, j=1, 2, . . . N has a spatiotemporalreceptive field described by the function D^(j)(x,y,t) 402, filtering avideo stream with the receptive field of the neuron j gives the outputv^(j)(t) 403, which can serve as the input to the TEM 404:

v ^(j)(t)=∫_(−∞) ^(+∞)(∫∫_(X) D ^(j)(x,y,s)I(x,y,t−s)dxdy)ds.   (32)

In some embodiments, the receptive fields of the neurons can have onlyspatial components, i.e. D^(j)(x,y,t)=D_(s) ^(j)(x,y)δ(t), where δ(t) isthe Dirac function.

Where t_(k) ^(j), k ∈

represents the spike time of neuron j, j=1, 2, . . . N, the t-transformcan be written as

${v^{j}\left( t_{k}^{j} \right)} = {\delta^{j} + {\sum\limits_{l < k}{{h\left( {t_{k}^{j} - t_{l}^{j}} \right)}.}}}$

In inner product form, this can be written as <I, ψ_(k) ^(j)>=q_(k) ^(j)with q_(k) ^(j)=δ^(j)+Σ_(l<k)h(t_(k) ^(j)−t_(l) ^(j)). The samplingfunction can be given by ψ_(k) ^(j)(x,y,t)={tilde over(D)}^(j)(x,y,t)*g(t−s_(k) ^(j)) where {tilde over(D)}^(j)(x,y,t)=D^(j)(x,y,−t). In embodiments with only spatialreceptive fields, the sample functions can be written as φ_(k)^(j)(x,y,t)=D_(s) ^(j)(x,y)*g(t−s_(k) ^(j)).

In some embodiments, as depicted in FIG. 5, decoding the TEM-encodedsignal can be recovered using the same frame ψ_(k) ^(j), j=1, 2, . . .N, k ∈

, with ψ_(k) ^(j)(x,y,t)=D(x,y,t)*g(t−s_(k) ^(j)) 503, 402. Where thefilters modeling the receptive fields D^(j)(x,y,t) are linearlyindependent and span the whole spatial domain of interest (i.e., forevery t₀ ∈

, Dj(x,y,t₀))_(j) forms a frame for L²(X), if the total spike densitydiverges in N, then there can exist a number

such that if N≧

then the video stream I=I(x,y,t) can be recovered as

$\begin{matrix}{{I\left( {x,y,t} \right)} = {\sum\limits_{j = 1}^{N}{\sum\limits_{k \in }{c_{k}^{j}{\psi_{k}^{j}\left( {x,y,t} \right)}}}}} & (33)\end{matrix}$

503 and c_(k) ^(j), k ∈

, j=1, 2, . . . , N, are suitable coefficients.

Letting [c^(j)]_(k)=c_(k) ^(j) and c=[c¹, c², . . . c¹]^(T), thecoefficients c can be computed as

c=G ⁺ q   (34)

502 where T denotes the transpose, q=[q¹, q², . . . q^(N)]^(T), [q^(j)_(k)=q_(k) ^(j) and G⁺ denotes the pseudoinverse. The entries of thematrix G can be given by

${G = \begin{bmatrix}G^{11} & G^{12} & \ldots & G^{1N} \\G^{21} & G^{22} & \ldots & G^{2N} \\\vdots & \vdots & \ddots & \vdots \\G^{N\; 1} & G^{N\; 2} & \ldots & G^{NN}\end{bmatrix}},$

[G^(ij)]_(kl)=<D^(i)(x,y,•)*g(•−t_(k) ^(i)), D^(j)(x,y,•)*g(•−t_(l)^(j))>. Where the receptive field is only in the spatial domain,

ψ_(k) ^(j)(t)=D _(s) ^(j)(x,y)g(t−s _(k) ^(j)) and [G^(ij)]_(kl)=(∫∫_(X) D ^(i)(x,y)D ^(j)(x,y)dxdy)g(t _(k) ^(i) −t _(l)^(j)).

In some embodiments, as depicted in FIG. 6, bounded and surjectiveoperations on the elements of a frame can preserve the frame'scharacteristics. As such, bounded and surjective operations can beperformed on the frames while in the spike domain (after being encodedby one or more TEMs and prior to being decoded by one or more TDMs) toalter the characteristics of the video. In accordance with the disclosedsubject matter, a method of dilating, translating, and rotating a visualstimuli can be achieved by applying, for example, the following functionto the encoded visual stimuli:

ψ_(α,x) ₀ _(,y) ₀ _(,θ)(x,y)=α⁻¹ R _(θ)ψ(a ⁻¹ x−x ₀ , a ⁻¹ y−y ₀),

601 where α represents the amount of dilation and is a real number notincluding zero; x₀, y₀ represents the amount of translation; θrepresents the amount of rotation between 0 and 2π, R_(θ)ψ(x,y)=ψ(xcos(θ)+y sin(θ),−x sin(θ)+y cos(θ)); and

${\psi \left( {x,y} \right)} = {\frac{1}{\sqrt{2\pi}}{\exp \left( {{- \frac{1}{8}}\left( {{4x^{2}} + y^{2}} \right)} \right)}{\left( {^{\; {kx}} - ^{k^{2}/2}} \right).}}$

In further embodiments, the alterations to the video can be achieved byusing a switching-matrix 701, as depicted in FIG. 7. To rotate the videoby an angle lθ₀, l ∈

, the spike coming from filter element ([x,y], α, θ) can be mapped tothe reconstruction filter at ([x,y], α, θ+lθ₀). To dilate, or zoom thevideo by a value α₀ ^(m), m ∈

, the spike coming from filter element ([x,y], α, θ) can be mapped tothe reconstruction filter at ([x,y], α₀ ^(m) α, θ). To translate a videoby [nb₀, kb₀], the spike coming from filter element ([x,y], α, θ) can bemapped to the reconstruction filter at ([x+nb₀, y+kb₀], α, θ). Tosimultaneous dilate by α₀ ^(m) and translate by [nb₀, kb₀], then thespike coming from filter element ([x,y], α, θ) can be mapped to thereconstruction filter at ([x+α₀ ^(m) nb₀, y+α₀ ^(m) kb₀], α₀ ^(m) α, θ).

The disclosed subject matter and methods can be implemented in softwarestored on computer readable storage media, such as a hard disk, flashdisk, magnetic tape, optical disk, network drive, or other computerreadable medium. The software can be performed by a processor capable ofreading the stored software and carrying out the instructions therein.

The foregoing merely illustrates the principles of the disclosed subjectmatter. Various modifications and alterations to the describedembodiments will be apparent to those skilled in the art in view of theteachings herein. It will thus be appreciated that those skilled in theart will be able to devise numerous techniques which, although notexplicitly described herein, embody the principles of the disclosedsubject matter and are thus within the spirit and scope of the disclosedsubject matter.

We claim:
 1. A multiple-input, multiple-output encoder comprising: afiltering kernel comprising a plurality (N) of sets of filters, each ofthe N sets of filters having a plurality (M) of filter elements; and atleast one time encoding machine (TEM), each of the M filter elements ofthe N sets of filter elements being additively coupled to acorresponding input of the at least one TEM.
 2. The encoder of claim 1wherein the at least one TEM comprises an integrate-and-fire neuron. 3.The encoder of claim 1 wherein the at least one TEM comprises a neuronhaving multiplicative coupling.
 4. The encoder of claim 1 wherein the atleast one TEM comprises an asynchronous sigma/delta modulator.
 5. Theencoder of claim 1 further comprising a plurality (N) of adders, eachadder coupled to a corresponding one of the N sets of filters and one ofa plurality (N) of bias values.
 6. The encoder of claim 1 wherein the atleast one TEM comprises an irregular sampler.
 7. The encoder of claim 1wherein the at least one TEM generates a spike sequence.
 8. The encoderof claim 1 wherein the at least one TEM comprises at least two ON-OFFneurons, each neuron being interconnected and having a feedbackmechanism.
 9. A decoder to decode a signal encoded by a multiple-input,multiple-output TEM encoder comprising: at least one Time DecodingMachine (TDM); and a filtering kernel comprising a plurality (N) of setsof filters, each of the N sets of filters having a plurality (M) offilter elements, a corresponding output of the at least one TDM beingcoupled to a corresponding one of the N sets of filters, each of the Mfilter elements of each set of filters being additively coupled to acorresponding M filter element of each other set of the N sets offilters.
 10. The decoder of claim 11, wherein the filtering kernel hasan overall response selected to at least substantially invert anencoding process of a TEM encoder used to encode the TEM-encoded signal.11. The decoder of claim 11, wherein the TEM-encoded signal isirregularly sampled.
 12. An encoder to encode a video stream signalcomprising: a filter to receive the video stream signal, the filterhaving a plurality (N) of filter elements; and a Time Encoding Machine(TEM) to generate a plurality (N) of TEM encoded spatiotemporal fieldsignals in response to a spatiotemporal field signal received from acorresponding one of the N filter elements.
 13. The encoder of claim 14,wherein the spatiotemporal field signals are described by an equation:v ^(j)(t)=∫_(−∞) ^(+∞)(∫∫_(X) D ^(j)(x,y,s)I(x,y,t−s)dxdy)ds, whereD^(j)(x,y,s) is a filter function, and I(x,y,t) represents the videostream signal.
 14. The encoder of claim 14 wherein the N TEM-encodedspatiotemporal field signals are represented by a sampling function:ψ_(k) ^(j)(x,y,t)=D(x,y,−t)*g(t−s_(k) ^(j)), for k spike times, for each(x,y) in a bounded spatial set, where j corresponds to each of the NTEM-encoded spatiotemporal field signals, and where g(t)=sin(Ωt)/πt. 15.A decoder to decode a TEM-encoded video stream signal comprising: a TimeDecoding Machine (TDM) to receive a plurality (N) of TEM-encodedspatiotemporal field signals, and for each TEM-encoded spatiotemporalfield signal generate a TDM-decoded spatiotemporal field signal; and anadder to combine each of said TDM-decoded spatiotemporal field signalsto recover the video stream signal.
 16. A system to alter a video streamsignal comprising: a plurality (N) of TEM-filters; a plurality (N) ofreconstruction filters; a switching matrix, operatively coupled to theplurality (N) of TEM-filters and the plurality (N) of reconstructionfilters, to map a plurality (N) of TEM-encoded spatiotemporal fieldsignals received from the N TEM-filters to the plurality (N) ofreconstruction filters in a video stream signal TDM.
 17. The system ofclaim 19, adapted to rotate the video stream signal, wherein theswitching matrix maps each of the N TEM-encoded spatiotemporal fieldsignals from a corresponding TEM-filter ([x,y], α, θ) to a correspondingreconstruction filter ([x,y], α, θ+lθ₀), where lθ₀ represents a desiredvalue of rotation.
 18. The system of claim 19, adapted to zoom the videostream signal, wherein the switching matrix maps each of the NTEM-encoded spatiotemporal field signals from a corresponding TEM-filter([x,y], α, θ) to a corresponding reconstruction filter ([x,y], α₀ ^(m)α, θ), where α₀ ^(m) represents a desired value of zoom.
 19. The systemof claim 19, adapted to translating the video stream signal by a value[nb₀, kb₀], wherein the switching matrix maps each of the N TEM-encodedspatiotemporal field signals from a corresponding TEM-filter ([x,y], α,θ) to a corresponding reconstruction filter at ([x+nb₀, y+kb₀], α, θ).20. The system of claim 19 to zoom the video stream signal by a value α₀^(m) and translate said video stream signal by a value [nb₀, kb₀],wherein the switching matrix maps each of the N TEM-encodedspatiotemporal field signals from a corresponding TEM-filter ([x,y], α,θ) to a corresponding reconstruction filter at {[x+α₀ ^(m) nb₀, y+α₀^(m) kb₀], α₀ ^(m) α, θ).